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SURVEY SYSTEM 

RELATED APPLICATIONS 

The present application is a continuation in part (CIP) of US patent application 
10/258,822, filed on October 25, 2002, which is a national phase of PCT patent application 
5 PCT/EL01/00391, filed on May 1, 2001, which claims the benefit under 119(e) of US 
provisional patent application 60/200,837, filed on May 1, 2000. The application also claims 
the benefit under 119(e) of US provisional patent application 60/424,328, filed on November 
7, 2002. The disclosure of all of these applications are incorporated herein by reference. 

FIELD OF THE INVENTION 
10 The present invention is related to the field of performing surveys, for example surveys 

in which the respondents discuss issues with each other. 

BACKGROUND OF THE INVENTION 
Surveying prospective buyers for information is an important tool of industry in 
deciding on investments, for example in new products, their packaging, pricing and 
15 components. Surveying is also used in other fields, for example in politics and employee 
satisfaction/feedback. 

Traditionally, surveys are performed by preparing a predetermined set of questions, 
selecting a group of respondents representing a population of interest and asking each of the 
people in the selected group, the questions in the predetermined set. The questions may be 

20 asked by a surveyor knocking on people's doors, in a shopping center, over the telephone 
and/or on a computer network. For example, US patent publication 2002/0128898, the 
disclosure of which is incorporated herein by reference, describes a method of assigning 
surveys to respondents, on the Internet. Surveys may also be provided to people gathered in a 
room to answer the questions. For example, when the survey requires presenting models to the 

25 respondents and/or requires expensive surveying apparatus, the respondents may be gathered 
for the survey in a specific location. 

Many surveys include closed questions in which the respondents need to select one or 
more of answers provided with the question or give a rating of a statement (e.g., 
agree/disagree, between 1 to 5). Closed questions allow fast analysis of the answers, and 

30 therefore collection of data from a large number of respondents, but do not provide for idea 

collection from the respondents. 

Some surveys include also open questions to which respondents answer in their own 

words. This allows for collecting ideas from the respondents but makes the analysis much 
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harder. Half open questions (referred to herein also as half closed), in which answers are 
provided in accordance with a given structure, for example the respondent needs to state a 
single word or fill in a detail having a known structure (e.g., a telephone number), serve as a 
compromise between open and closed questions. Half open questions include also questions in 
5 which the respondent may enter free text as a last resort, for example if none of the given 
options are suitable. 

A web site posted September 2001 describes an on-line survey system in which the 
answers of each of the respondents are displayed to the other respondents. The other 
respondents may vote for each answer using agree/disagree buttons. Thus, each answer 

10 provided by a respondent is immediately rated by the other respondents. 

Another method of performing market research involves conducting focus group 
discussions in which a plurality of respondents discuss issues brought up by a moderator. 
Focus group discussions allow group thought, providing a much more powerful method of 
collecting input from respondents than closed questions or even open questions. Focus group 

15 discussions, however, are generally limited to about ten to fifteen respondents, so that the 
discussion does not flood the group with answers and comments which cannot be 
comprehended by the moderator. U.S. patent 6,256,663 to Davis, the disclosure of which is 
incorporated herein by reference, describes a system for moderating an on-line focus group. 

In some cases, as in the '663 patent, a focus group over the Internet is conducted using 

20 a chat-room model in which all the users are connected to the Internet at the same time. In 
other cases, a focus group uses a message board model in which the respondents log in to the 
focus group in their free time and provide their comments. Use of a message board allows for a 
larger number of respondents, as the respondents have more time to read comments of others. 
However, a typical survey session according to this model has a duration of between about a 

25 few days to a few weeks. In this model there is also control over the survey and over the 
distribution of information. Thus, if large numbers x>f participants actually participate, the 
session may collapse under its own weight. 

In some cases, a market research is performed in two stages. In a first stage, a large 
number of respondents answer questionnaires without any interaction, in order to collect large 

30 amounts of quantitative data. In a second stage, a small group of respondents is asked to 
remain for a focus group, in order to perform a discussion, which better collects qualitative 
data. The focus group may be performed also before quantitative data is collected, in 
preparation for a large scale survey, for example as suggested in PCT publication 
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WO02/19232, the disclosure of which is incorporated herein by reference. In fact, some market 
research studies involve repeated interleaved quantitative surveys and focus group discussions 
until satisfactory results are achieved. This naturally is very costly and time consuming. 

A survey service provided by Axiom Group includes showing slides to a plurality of 
respondents who use buzzers to rate the products shown in the slides. Based on the results of 
one of the questions, a client may suggest to post another question. 

The above mentioned PCT publication WO02/19232 describes a system for conducting 
market research studies which includes a moderator computer and respondent computers 
connected over a network. The moderator directs the respondents to view preselected subject 
matter and collects reactions to the subject matter from the respondents. The reactions are 
described as including ratings and comments. The reactions are collected for later analysis, for 
example in order to build a large scale quantitative study. This PCT publication also includes a 
suggestion to perform studies on people located in a same room and after the reactions are 
collected perform an off-line discussion for additional information. The moderator may focus 
the respondents behavior, for example by forcing respondents to provide a comment before 
moving from one Web page to another. 

A company named "Informative" suggests an adaptive survey system in which after a 
question is posted with a plurality of possible answers, each respondent may select one of the 
answers or may suggest another answer which is added to the answers posted with the 
question. 

SUMMARY OF THE INVENTION 

A broad aspect of some embodiments of the invention relates to providing a moderator 
with tools for controlling interactive surveys, so as to achieve effective collection of 
statistically valid data. The tools are optionally adapted to control the survey session, in order 
to collect data based on statistical goals which are to be achieved. The control of the survey 
session may include, for example, the data exposure to respondents, which respondents take 
part and/or their biasing. 

A broad aspect of some embodiments of the invention relates to tools for controlling 

interactive surveys (i.e., surveys with interaction between the respondents), having large 

numbers of participants. Such tools optionally allow a moderator of the survey to control the 

people taking part in a portion of the survey, in real time. For example, people may be added 

as needed to achieve a certain goal, or be removed (e.g., from certain questions), for example 

to prevent them being wasted or biased. Alternatively or additionally, the tools aid the 
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moderator in assimilating large amounts of data, such that the data may be used during the 
survey session in directing the path of the survey. 

In an exemplary embodiment of the invention, the provided tools allow decisions to be 
made in real-time while the survey is going-on, for example allowing an operator to make 
5 changes, as necessary, in a preplanned survey plan. In some embodiments of the invention, the 
surveys are carried out without a living moderator, or with intermittent intervention of such a 
moderator. In an exemplary embodiment of the invention, the tools are used for qualitative 
surveys and/or focus groups. Alternatively or additionally, the tools are used for quantitative 
surveys and/or focus groups. In a survey, quantitative and qualitative events may occur in 

10 parallel or in series (e.g., in cycles). The presentation of a stimulus, for example a product 
description, may start a new cycle of qualitative and quantitative events. It is noted that in 
some embodiments of the invention, also qualitative information is treated as quantitative, for 
example for display as a summary. 

In some embodiments of the invention, the survey design includes enough slack to 

15 allow various mistakes to occur and still not prevent the final statistical validity of the results. 
In an exemplary embodiment of the invention, a pool of "spare" respondents is provided as 
one type of slack, for example, to allow additional respondents to be used for answering 
questions where a desired statistical confidence is not yet achieved. The bias of the 
respondents in this pool is optionally controlled. 

20 A broad aspect of some embodiments of the invention, relates to controlling and/or 

managing bias in a survey. In an exemplary embodiment of the invention, answers received 
from respondents are marked (and later analyzed) in a manner which reflects one or more bias 
factors of the respondent, for example, what stimuli the respondent was exposed to and/or a 
previous slant of answers of that respondent. Alternatively or additionally, bias is controlled by 

25 changing the order of questions and/or other stimuli (e.g. product descriptions, respondent 
comments) to one or more respondents. Alternatively or additionally, bias is controlled by 
changing other aspects of timing, for example, delay. Alternatively or additionally, bias is 
controlled by controlling a degree of interactivity of a focus group, for example to assist and/or 
prevent cementing of ideas by the respondents. Alternatively or additionally, bias is controlled 

30 by an automated tool suggesting to send or not send data and/or preventing the sending of data, 
to control data. 

A broad aspect of some embodiments of the invention, relates to context management. 

Typically, the context of a respondent indicates a bias of that respondent and/or may shade or 
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otherwise impact the true meaning of answers or remarks given by a respondent. The context 
may be represented, for example, by the stimuli (or some of the stimuli, for example only 
product pictures) displayed to the respondent and/or the order in which the stimuli was 
displayed. A moderator may have difficulty in dealing with too many contexts. In an 

5 exemplary embodiment of the invention, the proliferation of contexts is limited, for example, 
by limiting the number of possible contexts (e.g., by forcing respondents to see certain 
stimuli). Alternatively or additionally, a set of contexts may be predefined. Alternatively or 
additionally, contexts may be defined (ad hoc and/or to begin with) in a manner which allows 
combining the contexts, for example results from two similar (in a statistically meaningful 

10 manner or user designed manner) contexts may be combined into a single value for a summary 
display. 

In an exemplary embodiment of the invention, when an item of information is 
displayed, it is shown with a context, for example, a respondent answer is shown with a 
context of what stimuli the respondent was exposed to. Some actors, for example other 

15 respondents, may not be able to see some or all of such context information, for example, to 
reduce bias. When two contexts are combined, a manufactured context which indicates the 
common features of the contexts, is optionally shown. 

In an exemplary embodiment of the invention, a context includes one or more of 
stimuli that the respondent was exposed to, order of such stimuli, type(s) of interactions the 

20 respondent took part in and/or which other respondents the respondent interacted with. 

A broad aspect of some embodiments of the invention relates to automated supporting 
of a survey, for example to ensure the survey meets predetermined goals. Such support can 
include, for example, making suggestions to a moderator, preventing actions and/or creating 
questions. In an exemplary embodiment of the invention, an automated system is provided 

25 which monitors available resources (e.g., time, respondents, new concepts to be checked, 
moderator abilities, respondent abilities, stimuli and/or inducements) and makes suggestions 
how to apply the resources and/or whether one or more goals of the survey can be met. 
Optionally, the goals are prioritized and the system can plan resources for the goals, for 
example using planning methods well known in the art. 

30 An aspect of some embodiments of the invention relates to the provision of methods 

and apparatus for conducting interactive survey sessions, in which respondents provide 
answers to questions and discuss surveyed issues with other respondents, during the same 
session. 
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The answers to at least some of the questions are collected in a manner which allows 
quantitative analysis of the answers. The discussion, on the other hand, provides, for example, 
free text input from the respondents, similar to input received in qualitative research, such as a 
focus group discussion. Alternatively or additionally, the discussion is used for obtaining 
5 ratings of entries by other respondents. 

In an exemplary embodiment of the invention, quantitative data is collected by setting 
one or more statistical goals to be achieved and then controlling the survey so that these goals 
are achieved. In an exemplary embodiment of the invention, the statistical goals comprise 
achieving a proposition having a minimal agreement rate and/or a minimum statistical 

10 confidence level of the agreement rate. For example, one goal is to have at least one concept 
presented to be approved by at least x% of the respondents with at least y% confidence in the 
answer. In some cases an open-ended goal may be defined for example: a high as possible 
confidence level, but at least 75%. A more complex goal can also be defined, for example, that 
a certain type of statistical analysis may be performed at a later date. In one example, a 

15 plurality of sessions are set up (in parallel or consecutively), with each one being controlled so 
that the totality of sessions provide the desired goal and/or quantitative data. In an exemplary 
embodiment of the invention, what is required to achieve such confidence levels is a 
sufficiently large sample in which biases are controlled and/or prevented. Quantitative data, in 
general, is presented statistically, with confidence levels displayed or hidden. It is a feature of 

20 some embodiments of the invention, that also qualitative data is analyzed and/or displayed in a 
quantitative manner. 

In some embodiments of the invention, the answers to the questions are provided by 
the respondents intermittently with the discussion. At the end of the survey session, a report 
including quantitative data summarizing the respondents answers to questions, is optionally 

25 generated. The report may optionally be in the form of tables and/or charts. The report may 
also include remarks and/or other data from the discussion. In some embodiments of the 
invention, the report provides explanations given by the respondents to the quantitative data of 
the report. The explanations are optionally taken at least partially from the discussion. 
Alternatively or additionally, the explanations are given as responses to open questions. 

30 Optionally, the report further includes quantitative data on the explanations given by the 
respondents. 

In an exemplary embodiment of the invention, a report is displayed continuously or 

intermittently during the survey, for example to support real-time decision making, for 
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example, for deciding what to change in goals of survey, where to put effort, how to expose 
data (and to who), what was completed and/or what to use respondents for. In an exemplary 
embodiment of the invention, the report is displayed as compared to one or more goals that 
were set for the survey, for example before the survey and/or during the survey. Goals can be, 

5 for example, statistical goals (e.g., confidence level over 75% on which product design is 
better liked) and/or qualitative information collection goals (e.g., at least four different reasons 
that a product is preferred over competition). In an exemplary embodiment of the invention, 
the goals are shown in an updating report table, showing for example, goals, how close the 
goals are to being met and/or whether there appears to be a problem in meeting the goals. 

10 The term remarks is used herein to refer to respondent statements which are not 

answers to questions of the survey session. The remarks may relate to answers of other 
respondents, to questions of the survey and/or to remarks of other respondents or may be 
related to any other issue, such as to the quality of the stimuli or general remarks without 
relation to any specific statement. In some embodiments of the invention, at least some of the 

15 questions include closed and/or half closed questions which allow fast tabulation of the results. 

By performing the discussion and the collection of quantitative data in the same 
session, the feedback of the discussion and of the quantitative data collection may be used 
instantaneously during the session. This provides better chances of reaching conclusions in a 
shorter period. 

20 Interactive survey sessions are, typically, different from other sessions, such as on-line 

courses and presentations, in that survey sessions are intended primarily for collecting 
information rather than providing information. Therefore, for example, the respondents in a 
survey session are optionally selected based on statistical constraints so that the respondents 
properly represent a large group of interest. Alternatively or additionally, the questions 

25 provided to the respondents may be adjusted according to responses the respondents provided 
to previous questions. In addition, the flow of the session is optionally rigorously planned so 
that the session provides the desired information. The survey session may be performed for 
market research or for any other purpose. 

Optionally, the survey report of the quantitative data includes data from a large enough 

30 group of people, such that the data is statistically valid, e.g., a group with at least 30-50 
respondents, or even 100-200 respondents. In some embodiments of the invention, a qualified 
moderator and/or a staff of a moderator and observers may achieve statistically meaningful 
results even for 400-500 respondent or more, even many more. In an exemplary embodiment 
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of the invention, moderators are used mainly for managing the technical side of the survey 
(e.g., controlling bias, statistical confidence level) and the observers are used for managing the 
content side, for example, reviewing real-time results and deciding if to change the goals of the 
survey. However, the moderator can do both functions. Alternatively, other distributions of 
5 functions between a moderator, his assistants and/or observers, may be provided. 

In some embodiments of the invention, the survey report includes quantitative data 
from at least one additional survey session. Optionally, a plurality of survey sessions may be 
conducted at different times (or at the same time in parallel) using mostly the same or similar 
questions, in a manner which allows combining and/or comparing of the results or the survey 
10 sessions in generating a single report. 

In some embodiments of the invention, the discussion relates at least partially to the 
provided questions. Alternatively or additionally, one or more of the questions are generated 
based on information from the discussion or from the answers to other questions. 

In some embodiments of the invention, not all the respondents actively participate in 
15 the discussion, i.e., only answers and/or remarks of some of the respondents are provided to 
other respondents. Optionally, the number of respondents actively participating in the 
discussion is selected such that the amount of remarks provided to each respondent can be read 
and/or otherwise assimilated by the receiver. 

Alternatively or additionally, at least some of the respondents receive only some of the 
20 answers and/or remarks of the other respondents. Optionally, the respondents are divided into 
sub-groups, each respondent receiving answers and remarks only from the members in his/her 
sub-group. Alternatively or additionally, each respondent receives remarks and/or answers 
from other respondents, without relation to whether the other respondents receive remarks 
from him. 

25 In some embodiments of the invention, each respondent receives answers and remarks 

from a static group of respondents, for example selected at the beginning of the survey session. 
Alternatively, the rules governing the providing of the answers and remarks vary with the 
questions of the survey session and/or may be changed by the moderator. Optionally, the 
respondents are not directly notified whether their remarks and/or answers are provided to 

30 other respondents and/or to which other respondents. 

Optionally, the rules governing the providing of the answers and remarks depend on 

the profiles of the respondents. For example, respondents known to be individualists and/or of 

a high income may be given more remarks to view and/or their remarks may be provided to a 
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larger number of respondents. Further alternatively or additionally, the rules governing the 
providing of answers and remarks depend on the timing of the survey session in general or of a 
specific respondent. In some embodiments of the invention, the rules governing the providing 
of answers and remarks may depend on the remarks and answers, for example on the length 
5 and/or elaboration of the remarks, the time and/or speed in which they were typed by the 
respondent and/or whether specific keywords appear in the statements. 

In some embodiments of the invention, a single moderator controls the questions 
provided to all the respondents. Optionally, if desired, one or more observers aid the moderator 
in viewing the answers and remarks of the respondents and direct suggestions to the 

10 moderator. Alternatively or additionally, a plurality of moderators manage the survey session. 
Optionally, the moderators are assigned priorities which determine whose instructions prevail 
when contradicting instructions are received. 

Optionally, the same questions are provided to all the respondents. Alternatively, some 
of the questions are provided only to some of the respondents, for example questions directed 

15 only to respondents of a certain profile. Optionally, in order to save time, some questions may 
be provided only to a portion of the respondents, allowing the other respondents to perform a 
different task at the same time. 

Optionally, the moderator may state for each question a number of respondents to 
which it is to be directed and the survey system distributes the question accordingly. In an 

20 exemplary embodiment of the invention, such distribution takes into account desired statistical 
tests to be met, desired confidence levels and/or bias controlling issues. In some embodiments 
of the invention, the moderator may further state the profile of the people to receive a question 
and/or the percentage of people of a specific profile to receive the question. In some 
embodiments of the invention, at least some of the questions include closed questions and/or 

25 other questions which allow for collection of quantitative data. Optionally, at least some of the 
questions are provided to all the respondents. In some embodiments of the invention, the 
moderator states as detailed as possible rules before beginning the survey session. During the 
survey session, the moderator may optionally change the rules, if desired. 

The system of the present invention allows control of the number and identity of the 

30 people participating in a discussion, including conducting a plurality of discussions 

concurrently. The discussions are performed concurrently with the collection of data, allowing 

a moderator to use results from one of the quantitative data collection and the discussion in 

perfecting the other. The respondents in the discussions may be changed according to the 
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progression of the discussion and/or the results of the quantitative data collection. The control 
of the discussions is optionally performed in a manner which keeps all the respondents active. 
Optionally, respondents not required for discussion listen to the discussions of others and/or 
participate in less important discussions. 
5 An aspect of some embodiments of the invention relates to a computerized survey 

system in which the data displayed to respondents and/or the manner in which the data is 
displayed depends both on a profile of the respondent receiving the displayed data and on 
requirements on the number and/or profiles of respondents to receive the data. 

In some embodiments of the invention, the displayed data is selected during a survey 
10 session in which the data is displayed, optionally based on data which changes during the 
survey session. 

The displayed data may include questions from a moderator of the survey and/or 
automatically generated questions. Alternatively or additionally, the displayed data includes 
stimuli, answers of other respondents and/or remarks of other respondents. 

15 In some embodiments of the invention, the respondent profile includes a social 

characteristic of the respondent, for example whether the respondent is a leader or a follower, 
the reading rate of the respondent and/or any other attribute of the behavior and/or attitude of 
the respondent in survey sessions. Alternatively or additionally, the respondent profile includes 
demographic and/or geographic characteristics, such as gender, age and residence area. 

20 In some embodiments of the invention, the determination of the data displayed is 

performed responsive to actions and/or profiles of other respondents of the survey session. 

Optionally, the survey system determines which data is provided to each respondent 
and/or the timing of the display of the data. The timing may include, for example, the order of 
display, the duration of the display and/or the time allowed for response (if at all). The 

25 determination of which data is received by the respondent may depend on the people from 
which information is received (e.g., people of a specific attribute, people who have different 
tastes, a group to which the people belong), or on the data itself (e.g., short remarks, 
disagreeing remarks). In some embodiments of the invention, the data provided is selected at 
least partially according to the type of the question asked, e.g., the importance of the question, 

30 the relevance of the question to the respondent. 

In an exemplary embodiment of the invention, respondents identified as leaders are 

provided with answers of other respondents before answering a question, while a follower is 

provided with answers from others only after responding to questions. 
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The requirements on the number and/or profiles of respondents to receive the data 
optionally include requirements needed to ensure that the answers collected are statistically 
valid. 

An aspect of some embodiments of the invention relates to adjusting the data provided 

5 to a respondent during a survey session and/or the manner in which the data is displayed, 

according to the timing of the progression of the survey, e.g., the time passing from the 

beginning of the survey and/or the time remaining until the end of the survey. Optionally, the 

adjustment is performed by a survey pace keeper, who may be human and/or automated. 

Optionally, the remaining questions of the survey change according to the timing of the 

10 survey. Optionally, if the remaining time for answering the survey is limited, one or more 

questions are omitted. Alternatively or additionally, the number of possible answers is reduced 

from one or more questions. Further alternatively or additionally, open questions are converted 

to closed questions or vice versa according to the timing. Further alternatively or additionally, 

some or all of the questions are allotted specific time periods for responding, and the time 

15 allotted to answering one or more questions and/or the period in which the questions are 

provided are adjusted according to the timing of the survey. Further alternatively or 

additionally, the number of respondents to which questions are provided, is reduced. 

In an exemplary embodiment of the invention, the time for which data is exposed to the 

respondents is adjusted according to the time remaining until the end of the survey session. 

20 Alternatively or additionally, the time assigned to providing answers to closed questions and/or 

to providing ratings to answers of other respondents, is reduced. 

In some embodiments of the invention, the survey pace keeper is used with apparatus 

which provides the survey to a plurality of respondents substantially concurrently, and the 

change in the questions is performed in order to keep all the respondents in substantially the 

25 same pace, within a same time window or within a same discussion context. Alternatively or 

additionally, the change in the questions is performed in order to have the survey take up a 

predetermined amount of time without relation to the rate of answering of a specific 

respondent. This is optionally performed in a manner which does not impede the discussion 

interaction between the respondents. 

30 In some embodiments of the invention, the survey pace keeper provides suggestions on 

changes in the data provided to the respondents, to the moderator. Alternatively or 

additionally, the survey pace keeper automatically changes the data provided to the 

respondents, with or without the knowledge of the moderator. 
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An aspect of some embodiments of the invention relates to apparatus for automatically 
generating questions for a survey based on answers and/or remarks from a plurality of 
respondents to previously provided questions. The apparatus optionally receives sets of 
questions and answers and accordingly generates suggested questions. 
5 In some embodiments of the invention, the apparatus receives a question and a plurality 

of answers given to the question by respondents. The apparatus optionally selects from the 
received answers a plurality of answers to be included in a closed form of the question. The 
selected answers optionally include a predetermined number of most popular answers. 
Optionally, the apparatus also receives ratings given by the respondents to answers of other 
10 respondents. The closed question may include the answers having the highest ratings. 

Alternatively or additionally, the generated questions comprise open questions, which 
relate to answers of a plurality of respondents to one or more previous questions. The 
generated question may, for example, probe the reason why a certain answer has a high or low 
popularity. 

15 In some embodiments of the invention, the apparatus generates suggested questions for 

inclusion in the same survey session during which the answers are received. Alternatively or 
additionally, the questions are generated for a subsequent survey session. 

An aspect of some embodiments of the invention relates to apparatus for conducting a 
survey in which a respondent asked a question is provided with answers given by other 

20 respondents to the question, delayed relative to the time in which the answers were received 
from the other respondents. The delay is optionally used in order to prevent biasing of the 
respondent receiving the answers from other respondents. 

In some embodiments of the invention, the answers from the other respondents are 
displayed after the respondent answers the question. Optionally, the answers given by other 

25 respondents are collected from other respondents who are answering the survey substantially 
concurrently with the respondent to which the other answers are provided. In some 
embodiments of the invention, the answers provided to the respondent are selected based on 
one or more filtering parameters, for example, only short answers, are provided to the 
respondent. Alternatively or additionally, only answers disagreeing with the respondents 

30 answer and/or a certain percentage of disagreeing answers, are provided to the respondent. 

In some embodiments of the invention, the answers from the other respondents are 
provided after at least a predetermined time from displaying the question, so as to allow the 
respondent a period of thought. 
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In some embodiments of the invention, one or more answers are shown to the 
respondent, before the respondent provides an answer to the question. Optionally, the timing of 
the display of the answers of other respondents and/or the specific answers provided to the 
respondent depend on a profile of the respondent. 
5 An aspect of some embodiments of the invention relates to a system for conducting an 

interactive survey session by a single moderator who provides the questions, but not all 
respondents are provided the same question at the same time. During the interactive survey 
session, data is transferred between respondents, for example, the answers of at least some of 
the respondents are provided to others of the respondents. 
10 Optionally, different respondents receive the questions of the survey at different rates, 

for example according to the profiles of the respondents. Alternatively or additionally, 
different respondents receive the same questions in different orders, for example, for anti- 
biasing. 

In some embodiments of the invention, the respondents are divided into sub-groups 

1 5 which answer the questions in different orders. 

In some embodiments of the invention, a respondent may refer back to old questions 

during the interactive survey session. 

An aspect of some embodiments of the invention relates to a system for performing a 

survey, which includes a plurality of respondent stations and a plurality of control stations for 

20 controlling the data provided to the respondent stations. The plurality of control stations 

optionally may be used concurrently by different people. In some embodiments of the 

invention, the system includes a resolution unit, optionally included in one of the control 

stations, which determines what data is to be displayed when contradicting instructions are 

received from two different control stations. Optionally, the control stations have different 

25 priorities which determine the instruction that prevails in case of a contradiction. Alternatively 

or additionally, the contradicting instructions are provided to a third control station, which 

serves as an arbitrator. In some embodiments of the invention, the control of the data includes 

determining the questions displayed to the respondents and/or the timing of the display. 

In some embodiments of the invention, each of the respondent stations manages a 

30 buffer of data to be displayed. Data received by a respondent station from a control station is 

placed in the buffer and is displayed according to the order in which the data was received by 

the respondent station. Alternatively or additionally, one or more of the control stations has a 

higher priority and data from that control station is displayed before data from other stations. 
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An aspect of some embodiments of the invention relates to a survey system which 
provides at least some respondents with questions that they are not permitted to answer, but 
other respondents of the survey are allowed to answer. The display may be used to allow the 
respondents to know what the other respondents are relating to during a discussion between 
5 respondents. 

An aspect of some embodiments of the invention relates to a survey session which 
results in providing quantitative data on answers to both first and second questions, the second 
question relating to the reasons of the quantitative data on the answers of the first question. 
Optionally, the first and second questions are provided to the same respondents. Alternatively, 

10 the first and second questions are provided to partially overlapping groups of respondents. 
Optionally, the second question comprises an open question. Alternatively or additionally, the 
second question comprises a request to rank one or more reasons. The reasons may be, for 
example, generated by a moderator or may be collected from respondents by an open question. 
An aspect of some embodiments of the invention relates to a system for performing 

15 surveys, which monitors and/or analyses results substantially in real-time, with respect to goals 
of the survey. In an exemplary embodiment of the invention, if the results do not fit a goal, the 
system alerts a moderator that the goal was not achieved. In an exemplary embodiment of the 
invention, the goal is a threshold goal, for example that a confidence level is above a certain 
value or that at least a certain number of different answers are received. In an exemplary 

20 embodiment of the invention, the goal includes finding a single concept which has a distinctly 
better grade from a plurality of concepts. In some embodiments of the invention, the system 
suggests one or more questions to be used to probe for the reasons why the goal was not 
achieved. Alternatively or additionally, the system suggests or adds more respondents, if a 
statistical goal is not met. Further alternatively or additionally, the system suggests questions 

25 and/or stimuli to be used to achieve the goal. 

An aspect of some embodiments of the invention relates to using ratings from one or 
more respondents to assist text mining. In an exemplary embodiment of the invention, a 
system is provided for identifying similar statements, for example equivalent answers to a 
question. The system optionally receives a plurality of statements and provides the statements 

30 to a plurality of respondents for rating. For each pair of statements, a rating similarity score is 
defined based on the similarity of the ratings of the statements. In addition, the statements are 
assigned a text similarity score based on text mining methods. A total similarity score is 
optionally provided as a function of the text similarity score and the rating similarity score. 
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An aspect of some embodiments of the invention, relates to survey design. In an 
exemplary embodiment of the invention, a survey is designed to include focus groups that 
provide statistically valid data, by setting goals which are to be met, for example qualitative 
and/or quantitative goals. Optionally, the design includes defining one or more questions, sets 
5 of questions, question templates, contexts, types of discussion, groups, sub-groups, data 
exposure rules, stimuli, questions and/or stimuli orders and/or slack in a survey. Alternatively 
or additionally, the design includes defining a series of survey sessions which together 
cooperate to provide the desired information. In an exemplary embodiment of the invention, 
the design takes into account a large number of respondents, for example, 40, 60, 100, 200, 

10 500, 1000 or more. 

There is therefore provided in accordance with some embodiments of the invention, a 
method of conducting a survey, comprising providing a plurality of questions to respondents, 
at least some questions being provided to a plurality of the respondents, collecting answers to 
the questions from the respondents, transferring at least some of the answers to respondents 

15 other than those who generated the answers, collecting responses to the transferred answers, at 
least some of the responses being qualitative, and providing a statistical report which is at least 
partially based on qualitative responses collected from the respondents. 

Optionally, collecting the responses comprises collecting from fewer than all the 
respondents receiving the questions. Optionally, the plurality of respondents comprise at least 

20 30 respondents. Optionally, collecting responses to the transferred answers comprises 
collecting remarks. Optionally, the method includes using the statistical report in providing 
one or more of the questions. Optionally, providing the statistical report comprises generating 
the report automatically. Optionally, providing the questions, collecting the answers and 
responses and transferring the answers are performed during a time bound survey session. 

25 Optionally, the time bound survey session has a duration of less than three hours. Optionally, 
at least one of the questions is provided after transferring at least some of the answers to 
respondents other than those who generated the answers. Optionally, substantially all the 
questions are provided to substantially all the respondents. 

Optionally, at least one of the questions is provided to fewer than all the respondents. 

30 Optionally, the respondents are divided into sub-groups and wherein transferring the answers 

comprises transferring each answer to respondents in the same sub-group as the respondent 

generating the answer. Optionally, at least one of the provided questions is generated 

responsive to the collected responses. Optionally, at least one of the provided questions is 

15 
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generated responsive to statistics on the answers collected from a plurality of respondents. 
Optionally, the at least one question is generated automatically. Optionally, transferring the 
answers comprises transferring answers from at least one of the respondents to a group of 
respondents which changes during the survey for different answers of the respondents, 
5 Optionally, the group of respondents receiving the answers changes at least partially according 
to the contents of the answers. Optionally, the group of respondents receiving the responses 
changes at least partially according to a question with which the remark is associated. 

There is further provided in accordance with some embodiments of the invention, a 
system for conducting a survey, comprising a plurality of respondent stations adapted to collect 

10 statements from respective respondents, at least one presentation station adapted to provide 
questions to the respondent stations for display to the respondents, a server adapted to provide 
statements collected by at least one first respondent station to one or more second respondent 
stations, at least one of the collected statements being generated responsive to a statement 
provided by the server, and a report unit adapted to provide a report of quantitative data 

15 summarizing answers provided by the respondents to provided questions. 

Optionally, the respondent stations are adapted to collect each statement with relation 
to a specific provided question or statement. Optionally, the server and at least one 
presentation station are implemented by a single computer. Optionally, the at least one 
presentation station is adapted to collect answers to the questions. Optionally, at least some of 

20 the questions include closed questions. 

There is further provided in accordance with some embodiments of the invention, a 
method of conducting a survey with a plurality of respondents, comprising determining a 
profile of at least one respondent, receiving data to be distributed to at least some of the 
respondents, providing at least one requirement on the number or profiles of the respondents to 

25 receive the data; and distributing the data to at least some of the respondents in a manner 
selected responsive to the profile and the at least one requirement. 

Optionally, receiving the data comprises receiving a question. Optionally, receiving the 
data comprises receiving a remark or answer from one of the respondents. Optionally, the 
method includes receiving responses from the respondents and wherein receiving the data 

30 comprises receiving data generated responsive to the responses from the respondents. 
Optionally, determining the profile comprises determining a demographic or geographic 
characteristic of the respondent. Optionally, determining the profile comprises determining a 
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social characteristic of the respondent Optionally, providing the at least one requirement 
comprises providing a requirement on the number of respondents to receive the data. 

Optionally, providing the at least one requirement comprises providing a requirement 
on the percentage of respondents having a specific characteristic, to receive the data. 
5 Optionally, distributing the data to at least some of the respondents comprises distributing to 
respondents selected responsive to the profile and the at least one requirement. Optionally, 
distributing the data to at least some of the respondents comprises distributing to respondents 
selected responsive to the generator of the distributed data. Optionally, distributing the data to 
at least some of the respondents comprises distributing to respondents selected responsive to 

10 the contents of the distributed data. Optionally, distributing the data to at least some of the 
respondents comprises distributing the data with a timing selected responsive to the profile and 
the at least one requirement. Optionally, distributing the data with a timing selected responsive 
to the profile and the at least one requirement comprises distributing answers to a question to 
at least some of the respondents, after they provide answers to the question. 

15 There is further provided in accordance with some embodiments of the invention, a 

method of conducting a survey, comprising receiving data on the timing of the progression of 
the survey; and displaying data of the survey to a respondent in a manner selected responsive 
to the received timing data. 

Optionally, the received data on the timing of the progression of the survey comprises 

20 receiving data on the time passing from the beginning of the survey or remaining until the end 
of the survey. Optionally, displaying data of the survey to the respondents in a manner selected 
responsive to the received timing data comprises selecting the wording of one or more 
questions of the survey responsive to the timing data. Optionally, selecting the wording of one 
or more questions of the survey comprises selecting whether to display a closed, open or half 

25 closed question. Optionally, the method includes preparing a roster of questions to be asked 
during the survey and wherein displaying data of the survey to the respondents in a manner 
selected responsive to the received timing data comprises deterring from the roster of 
questions responsive to the received timing data. Optionally, deterring from the roster of 
questions comprises skipping at least one question in the roster. Optionally, deterring from the 

30 roster of questions comprises reducing the number of answers in at least one question of the 
roster. 

Optionally, displaying data of the survey to the respondents in a manner selected 

responsive to the received timing data comprises setting a time for responding to the displayed 
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data responsive to the timing data. Optionally, the survey has a plurality of concurrently 
connected respondents and wherein displaying data of the survey to a respondent in a manner 
selected responsive to the received timing data comprises displaying such that all the 
respondents are within a same context of the survey. 

There is further provided in accordance with some embodiments of the invention, a 
survey system, comprising a question provider adapted to provide a question to a plurality of 
respondents, an answer input unit adapted to receive answers from the respondents and a 
processor adapted to generate at least one question, responsive to the answers received from 
the respondents. 

Optionally, the processor is adapted to select from a plurality of answers given to an 
open question, several answers to be used in a closed question version of the open question. 
Optionally, the answer input unit is adapted to receive from the respondents ratings of the 
answers, and wherein the processor uses the ratings in generating the at least one question. 
Optionally, the processor is adapted to generate open questions. Optionally, the processor is 
adapted to generate an open question which asks for the reason of the statistical distribution of 
answers to one or more questions. Optionally, the question provider is adapted to provide at 
least one question generated by the processor during a same survey session in which the 
answers used in generating the question were received. 

There is further provided in accordance with some embodiments of the invention, a 
method of conducting a survey, comprising providing a question to a plurality of respondents, 
receiving answers from at least some of the respondents; and providing a first respondent with 
an answer received from at least one second respondent, delayed relative to the time the 
answer was received. 

Optionally, providing the answer received from the at least one second respondent 
comprises providing the answer from the second respondent only after receiving an answer 
from the first respondent, although the answer from the second respondent was received before 
the answer from the first respondent. Optionally, providing the first respondent with an answer 
from at least one second respondent comprises providing the first respondent with a plurality 
of answers. Optionally, providing the first respondent with an answer from at least one second 
respondent comprises providing the first respondent with fewer than all the answers received 
from other respondents. Optionally, providing the first respondent with an answer from at least 
one second respondent comprises providing one or more answers which fit one or more 
filtering requirements. 
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There is further provided in accordance with some embodiments of the invention, a 
method of conducting an interactive survey session, comprising providing at least one question 
to a plurality of respondents during the session, conducting a discussion between at least some 
of the plurality of respondents during the session, and providing at least two of the plurality of 
respondents with different questions, substantially concurrently during the session. 

Optionally, conducting the discussion comprises passing at least one statement 
generated by one of the respondents to substantially all the other respondents. Optionally, 
providing at least two of the respondents with different questions comprises providing a first 
respondent with a question not provided to a second respondent throughout the session. 
Optionally, providing at least two of the respondents with different questions comprises 
providing a first respondent with a question provided to a second respondent earlier during the 
session. 

There is further provided in accordance with some embodiments of the invention, a 
survey system, comprising a plurality of respondent stations adapted to provide questions to 
respondents and at least two control stations adapted to control the provision of data by the 
respondent stations. 

Optionally, the system includes an arbitration unit adapted to resolve contradicting 
instructions received from the at least two control units. Optionally, a first control station 
controls a first group of respondent stations and a second control station controls a second 
group of respondent stations different from the first group. Optionally, the first control station 
controls at least one of the respondent stations of the first group to display data received from a 
respondent station of the second group. Optionally, the at least two control stations control the 
data provided to the respondent stations. Optionally, the at least two control stations control 
the timing of the display of the respondent stations. 

There is further provided in accordance with some embodiments of the invention, a 
survey system, comprising a display unit adapted to provide questions to a respondent, and an 
input interface adapted to receive answers to the questions from the respondent, the input 
interface does not accept an answer for at least one of the displayed questions. Optionally, the 
display unit is adapted to provide the respondent with answers to the question for which the 
input interface does not accept answers. Optionally, the input interface is adapted to accept 
responses to the question for which the input interface does not accept answers, from the 
respondent. 
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There is further provided in accordance with some embodiments of the invention, a 
method of conducting an interactive survey session, comprising providing, during a survey 
session, a first question to a first group of a plurality of respondents, collecting answers to the 
first question, statistically analyzing the collected answers to the first question, providing, 
5 during the survey session, a second question on the reasons to the statistical distribution of the 
collected answers to the first question, to a plurality of respondents, collecting answers to the 
second question, and providing a statistical tabulation of the answers to the second question. 

Optionally, the second question comprises an open question. Optionally, the second 
question comprises a request to rate one or more given explanations to the statistics of answers 
10 of the first question. 

There is further provided in accordance with some embodiments of the invention, a 
method of conducting a survey, comprising setting a survey goal, providing a plurality of 
questions to respondents, collecting answers to the questions from the respondents, 
transferring at least some of the answers to respondents other than those who generated the 
15 answers, comparing a current state of the survey to the goal, and controlling the transferring of 
the answers responsive to the comparison. 

Optionally, controlling the transferring of the answers comprises determining a number 
of respondents to receive an answer. Optionally, transferring at least some of the answers 
comprises transferring to fewer than all the respondents. 
20 There is further provided in accordance with some embodiments of the invention, a 

survey system, comprising a memory for storing a survey goal, a data provider for providing 
data to respondents, an input unit adapted to receive input from the respondents, responsive to 
the provided data, and a processor adapted to analyze the input from the respondents in order 
to determine a survey state, to compare the survey state to the survey goal and to control the 
25 providing of data to the respondents responsive to the comparison. 

Optionally, the data provider is adapted to provide questions to the respondents. 
Optionally, the processor is adapted to control the number of questions provided to the 
respondents responsive to the comparison. Optionally, the processor is adapted to determine 
when to terminate the survey responsive to the comparison. Optionally, the data provider is 
30 adapted to transfer input received from respondents to other respondents that did not provide 
the input. Optionally, the data provider is adapted to transfer at least some of the input received 
from respondents to fewer than all the other respondents that did not provide the input. 
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Optionally, the processor is adapted to control the number or identity of respondents to which 
the input is transferred, responsive to the comparison. 

Optionally, the input unit is adapted to receive from the respondents answers to 
questions. Optionally, the processor is adapted to determine a binary value as to whether the 
5 goal was achieved responsive to the comparison. Optionally, the processor is adapted to 
determine an extent to which the survey state is distanced from the goal responsive to the 
comparison. Optionally, the input unit is adapted to receive remarks and/or answers from the 
respondents. 

BRIEF DESCRIPTION OF THE FIGURES 

10 Particular, non-limiting embodiments of the invention will be described with reference 

to the following description of embodiments, in conjunction with the figures, in which: 

Fig. 1 is a generalized block diagram of a survey system, in accordance with an 
embodiment of the invention; and 

Fig. 2 is a flowchart of a survey session, in accordance with an embodiment of the 
15 present invention. 

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS 
Fig. 1 is a generalized block diagram of an on-line survey system 100, in accordance 
with an embodiment of the invention. A plurality of respondents 102 connect, for example 
using computers 104, to a presentation station 110, which is used to manage a survey. In an 
20 exemplary embodiment of the invention, computers 104 connect to presentation station 110 
through a communications medium 106 (e.g., an Internet). 

Presentation station 110 optionally provides questions and stimuli to respondents 102 
via computers 104 and receives responses from respondents 102. Alternatively or additionally, 
stimuli may be provided to the respondents using other methods, such as mail and cable TV. 
25 Presentation station 110 optionally governs the display format on computers 104 and the 
method of allowing input of data from respondents 102. Optionally, each computer 104 
operates a web browser which runs file received from presentation station 110. Alternatively 
or additionally, a dedicated software on computers 104 is controlled by presentation station 
110. 

30 Alternatively or additionally to computers 104, respondents 102 and/or moderator 128 

may interact with the survey using other communication devices, such as cellular telephones, 
interactive televisions, and/or personal data assistants (PDAs), such as Palm Pilots or Visors. 
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In some embodiments of the invention, presentation station 110 operates survey 
sessions automatically based on instructions preprogrammed into presentation station 110, for 
example by a researcher. Alternatively or additionally, a moderator 128 optionally controls 
presentation station 1 10 during survey sessions, for example by stating questions and stimuli to 
be provided to respondents 102 and/or controlling to which of the respondents the questions 
are directed. The stimuli provided during the survey session is optionally prepared in advance 
before the survey session. Alternatively or additionally, moderator 128 may prepare stimuli 
during the survey session. In some embodiments of the invention, presentation station 110 
displays responses received from respondents 102 to moderator 128. Optionally, presentation 
station 110 transfers responses from respondents 102 to other respondents as described below. 
The transfer of responses is optionally performed under general instructions of moderator 128 
during survey sessions and/or based on previously programmed instructions. The general 
transfer instructions may include, for example, a statement on the number of respondents to 
which specific responses are to be transferred and/or the identities of the respondents to which 
responses from a specific respondent are to be transferred. 

In some embodiments of the invention, one or more observers 136, using observer 
stations 138, view the flow of a survey session. Optionally, observers 136 may send 
suggestions regarding the flow of the session to moderator 128. Alternatively or additionally, 
observers 136 may interrupt in one or more parameters of the control of the survey session 
and/or may override moderator 128. Observer 136 may optionally be a client interested in 
viewing the survey session in real-time. Alternatively or additionally, observer 136 is a 
moderator assistant who helps the moderator by summarizing data generated by the 
respondents in the survey session. In an exemplary embodiment of the invention, the 
moderator's task is limited to technical management of the survey session, while the data 
review is performed only by the observers. 

In some embodiments of the invention, observers 136 control their observer stations 

138, determining what data to view and how the data is presented. Alternatively, observers 136 

are presented the same data as presented to moderator 128 or are presented with other data 

whose contents is determined by moderator 128, from presentation station 110. Further 

alternatively or additionally, observer stations 138 may be controlled by moderator 128 or 

observer 136. Observer stations 138 maybe computers or any other communication apparatus, 

such as mentioned above. In an exemplary embodiment of the invention, one or more observer 

stations 138 comprise cellular phones on which respondent remarks are sounded and/or 
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displayed. Suggestions to moderator 128 may be transmitted in SMS messages to presentation 
station 110. 

In some embodiments of the invention, system 100 allows a single moderator to 
manage a survey session of many respondents, e.g., hundreds or even thousands, in which the 
5 respondents conduct discussions related to the survey. 

Reference is also made to Fig. 2, which is a general flowchart of a survey session, in 
accordance with an exemplary embodiment of the present invention. It is noted, however, that 
the survey may have other structures, for example a more free discussion with fewer questions. 
Before the session begins, a researcher optionally prepares (202) a roster of questions which 

10 are to be provided to respondents 102. The questions may be open questions, half open 
questions and/or closed questions. The questions may be displayed to the respondents and/or 
sounded to the respondents. In some embodiments of the invention, one or more of the 
questions may be accompanied by images, video clips, audio files and/or any other stimuli. 
The term question refers herein also to statements and other stimuli which arouse responses, 

15 even if they are not linguistically defined as questions. 

In some embodiments of the invention, the roster may include optional questions 
and/or display modes which may be used by moderator 128 during the survey session. 
Optionally, the roster may be built in the form of a flowchart with decision points along the 
path. During the survey session, moderator 128 decides which path to use and accordingly the 

20 survey proceeds. Alternatively or additionally, at one or more decision points, presentation 
station 1 10 selects the path taken according to one or more parameters of the survey session. 
The roster may also include other parameters of the session, such as the stimuli to be provided 
along the survey session and/or the numbers and/or identities of the respondents of each 
question. 

25 In some embodiments of the invention, the questions of the roster are organized in 

context groups, the questions of each group relating to a respective context, for example 
different concepts and/or ideas, such as product packages and/or names. Optionally, along with 
each question, respondents 102 are displayed a stimuli relating to the context. 

In an exemplary embodiment of the invention, each question is assigned an 

30 identification number (ID) (and/or name) which allows easy tracking of the questions, for 
example in comparing to answers of same questions in other sessions. Optionally, the question 
IDs include a complex structure which identifies the context of the question and the type of the 
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question. This complex structure allows easy comparison of same questions relating to 
different concepts. 

In some embodiments of the invention, moderator 128 and/or a researcher prepares 
(203) the session, for example by recruiting people to participate in the survey session, setting 
5 the time of the session and making sure all respondents have required hardware and/or 
software. During the survey session, presentation station 110, automatically and/or under 
instructions of moderator 128, presents (204) the respondents 102 with questions from the 
roster and/or with questions generated in real time. Respondents 102 enter (206) answers to the 
questions to their computers 104. These answers are transmitted to presentation station 110, 

10 which processes the responses and provides (210) an answer summary to moderator 128. 
Alternatively or additionally, presentation station 110 provides (212) a complete record of the 
answers to moderator 128 and/or to observers 136. In some embodiments of the invention, 
observers 136 provide important excerpts and/or suggestions to moderator 128. Further 
alternatively or additionally, a complete answer record is provided for a selected group of 

15 important questions and/or from a selected sub-group of respondents 102 whose answers are 
expected to be of particular on-line interest. 

In some embodiments of the invention, presentation station 110 generates (214) 
additional questions responsive to the received responses. The generated additional questions 
are optionally provided to moderator 128 who may determine if and when to provide the 

20 generated question to the respondents. Alternatively or additionally, generated questions may 
be provided automatically by station 1 10 based on previously configured rules and/or decisions 
made during the survey session. 

In some embodiments of the invention, answers to open questions and/or half open 
questions received from respondents 102 are provided (216) to other respondents for viewing, 

25 in order to receive rating and/or arouse a discussion between the respondents. The respondents 
receiving answers from other respondents may optionally use these statements in preparing 
their answers (206) to the same question and/or may respond (218) to these answers with 
discussion remarks.. The discussion remarks are optionally provided to other respondents for 
review and discussion. In some embodiments of the invention, answers from other respondents 

30 may be presented as questions to which the other respondents may provide ratings (e.g., 

agree/disagree, between 1-10) and/or statement answers. 

The interaction between the respondents may be at different levels, for example based 

on a moderator selection. The selection may be based, for example, on the amount of ideas it is 
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desired to get from the respondents, versus the time constraints of the survey session. 
Optionally, the interaction between respondents may include rating of answers of other 
respondents, providing remarks to answers of other respondents, one sided transfer of remarks, 
open discussion (direct or through the moderator) and/or allowing transfer of a limited number 
5 of remarks. 

Alternatively to presentation station 110 collecting answers and transmitting the 
answers to computers 104, each computer 104 is instructed to which other computers to 
transmit its answers. Alternatively or additionally, a router and/or server connected before 
presentation station 110 handles the answer transmission, so as to reduce the load on 

10 presentation station 1 10. The server may optionally perform other tasks described hereinbelow 
as being performed by presentation station 1 10, for example data buffering. 

Moderator 128 optionally follows the discussion and decides (220) when to end the 
period allotted to answering the current question. If (222) an additional question is to be 
provided, moderator 128 selects (224) an additional question to be provided to respondents 

15 102. At the end of the survey session, presentation station 110 optionally generates (226) a 
session report. 

Computers 104 are optionally adapted to keep respondents 102 busy to a sufficient 
extent. Optionally, presentation station 110 provides each respondent 102 with a sufficient 
amount of data, such that the respondent is not bored. In some embodiments of the invention, 

20 data is provided to each respondent 102 at a predetermined rate, for example a fixed number of 
characters or statements per minute. The data may include, for example, questions, answers 
from other respondents and/or remarks. 

Referring in more detail to preparing (202) the roster of questions, the roster of 
questions maybe generated for a single subject or for a plurality of different subjects. In some 

25 embodiments of the invention, a single set of questions is prepared for a plurality of sessions. 
Presentation station 110 is optionally adapted to shuffle the questions in the roster, so that 
different question orders are used for different sessions, to prevent bias. In some embodiments 
of the invention, the shuffling is performed randomly for each session. Alternatively, a 
sequence of sessions is pre-planned with each session having a different arrangement of 

30 questions. Further alternatively or additionally, the shuffling is performed based on results of 
previous sessions. Optionally, the set of questions may be adjusted based on discussions 
during previous sessions. 
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In some embodiments of the invention, the results of a previous survey session are used 
in generating questions of the roster. For example, an open answer provided in a previous 
session may be converted into a closed question or a half closed question based on answers 
received in the previous session. 

Optionally, one or more open questions in the roster, prepared before the session, are 
associated with possible answers, for example answers from previous sessions. In some 
embodiments of the invention, if a sufficient number of answers are not received from the 
respondents during the survey session, the answers from the roster are provided to the 
respondents for rating as if they were received from respondents of the current session. 

In some embodiments of the invention, the roster of questions includes for each 
question, a period of time during which the question is to be displayed and/or a stimuli 
sequence to be shown before and/or along with the question. A time for receiving answers may 
also be stated for some or all of the questions. 

In some embodiments of the invention, the roster of questions includes more questions 
than are planned to be used to allow moderator 128 to select questions according to the 
development of the session. Optionally, the roster is organized in the form of a tree with 
different branches according to the direction the discussion follows. 

It is noted, however, that in many cases it is hard to predict which questions will be 
relevant, before the discussion begins. Therefore, in some embodiments of the invention, 
moderator 128 is able to use questions not in the roster, modify questions from the roster 
and/or not use questions although they are included in the roster. Optionally, the roster 
includes question templates which allow fast generation of questions by the moderator during 
the survey session. An exemplary template may have the form "why do members of 'group V 
prefer this product in comparison to members of 'group 2'?", or the form "which of the 
following products 'product 1 ', 'product T . . . 'product n' would you prefer to buy?". 

Referring in more detail to preparing (203) the survey session, in some embodiments of 
the invention, the selection of the respondents is performed according to statistical parameters, 
so that the respondents truly represent a population of interest. Optionally, respondents are 
offered compensation for participating in the survey session. In some embodiments of the 
invention, the extent of the compensation varies with the correlation of the respondent to a 
desired portfolio and/or with the extent to which the respondent cooperates, answers questions 
and/or actually assists statistically (e.g., adds to the confidence of the results, reduces 
variance). 
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The number of respondents is optionally relatively large (e.g., above 50, 100, 300, 700 
or more) in order to provide statistical authenticity to the survey results. Optionally, in 
providing the answers between respondents, each respondent receives a limited number of 
answers, so that respondents are not flooded with information, as discussed below. 
5 In some embodiments of the invention, when too many people come to a survey and/or 

people not fitting the survey ask to participate, these people are rejected from participating in 
the survey. Alternatively, these people are given a different survey, for example one which is 
not required, in order not to discourage them from coming again to surveys or to keep them in 
standby in case they are needed during the survey session. In some embodiments of the 

10 invention, rather than answering a totally unrelated survey, the excess respondents may be used 
as a control group or for other less important tasks. 

Referring in more detail to respondents 102 entering (206) answers to the questions, in 
some embodiments of the invention, the answers and remarks are entered in text, for example 
using a keyboard. Alternatively or additionally, respondents 102 may speak out their answers 

15 and computer 104 converts the answers into text. Alternatively or additionally, computer 104 
transmits voice files with the respondents answers to presentation station 110, where the 
answers are converted into text, for transfer to other respondents and display to moderator 128. 
Optionally, moderator 128 may request to hear one or more respondent remarks, for example 
after the survey session is completed. In some embodiments of the invention, computer 104 is 

20 attached to video apparatus (not shown) which is used to collect additional information on the 
respondent's attitude toward the discussed subjects. Further alternatively or additionally, 
answers may be provided as drawings and/or other non-text visual input. 

The answers provided by the respondents may be of one or more types, including, for 
example, selecting one or more answers of closed questions, Providing a remark regarding the 

25 selection, providing a rating of a statement, providing a word and/or providing a text 
statement. Optionally, the questions provided by the moderator define which of the answer 
methods may be used. 

In some embodiments of the invention, a respondent may provide answers to old 
questions that he/she did not answer earlier, may provide additional answers to old questions 

30 and/or may replace old answers. Optionally, when a respondent refers to an old question (e.g., 

by scrolling or using a back button), the stimuli associated with the old question are provided 

again to the respondent. The answer to the old question is optionally provided to presentation 

station 110 along with an indication on the information provided to the respondent until the 
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answer was provided. This allows presentation station 110 to indicate a bias factor for the 
answer. 

In some embodiments of the invention, while a respondent is referring to an old 
question of a different context, the respondent is not provided with questions and/or remarks 
5 from other respondents. After returning to the current context, the respondent is provided with 
the questions and remarks that he/she did not receive. Alternatively, the respondent is provided 
with fewer questions, in order to allow the respondent to catch up with the other respondents. 
Alternatively, the respondent is allowed to continue participating in his current discussion 
while referring to an old question. In some embodiments of the invention, when a respondent 
10 relates to an old question of a context which is currently being discussed by a different group 
of respondents, the respondent is joined into this group instead of, or in addition to, his current 
group. 

Optionally, the moderator may control the ability of respondents to refer back to old 
questions. The control is optionally performed through presentation station 110, by stating 

15 general rules and/or specific instructions (e.g., relating to a specific respondent). The 
moderator can control, for example, for each respondent and/or for each question, how much 
back the respondent may refer, if at all. For example, after a certain amount of data is viewed, 
respondents may be prevented from referring back to some questions. Alternatively or 
additionally, respondents who are slow in answering questions may be prevented from 

20 referring back to old questions, so that they do not waste time on these questions instead of 
answering more questions. In some embodiments of the invention, when important questions 
are displayed, respondents may not refer to other questions. 

Referring in more detail to providing (210) an answer summary to the moderator, in 
some embodiments of the invention, the summary states, for closed questions, the percentage 

25 of respondents agreeing with each possible answer. In some embodiments of the invention, 
each respondent is assigned a weight which is given to answers provided by the respondent, 
and the summary is adjusted according .to the weights. Thus it is possible to correct for under- 
representation of certain groups (e.g., demographic and/or geographic groups). In some 
embodiments of the invention, different weights are given to responses given by respondents 

30 who saw responses of other respondents before answering. 

Optionally, each question is marked with the number of respondents to which the 
question was provided, a group profile of the respondents (e.g., 60% men, 40% women and/or 
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sub-groups to which they belong) and/or the number of respondents providing answers to the 
question. 

In some embodiments of the invention, for questions whose answers range on a scale, 
an average or other score value is used to represent the answers given by the respondents. 

For open questions, the summary optionally groups similar answers together and states 
for each answer the percentage of respondents providing the answer. In some embodiments of 
the invention, text mining methods are used to analyze the answers. For example, dominant 
words are extracted from the answers by presentation station 1 10 and the number of answers 
including each dominant word are included in the summary. Moderator 128 may optionally 
request to view the answers including a specific dominant word. 

Optionally, the summary differentiates between answers given by respondents having 
different contexts. For example, the answers of respondents seeing first and second products 
may be displayed separately from answers of respondents seeing second and third products. 

In some embodiments of the invention, text mining methods are used to identify 
positive and/or negative words. Answers including negative paradigms are optionally grouped 
together. Alternatively or additionally, the answers are displayed with negative words marked. 

As described below, in some embodiments of the invention, the respondents rate their 
agreement with answers of other respondents. Optionally, these ratings are used together with 
text mining in combining similar answers. In some embodiments of the invention, answers are 
combined as being similar if they have a similar rating and similar text mining results. 
Alternatively or additionally, answers are considered opposites if they have similar text mining 
results except for negation words and they have substantially opposite rating results. 

In some embodiments of the invention, the ratings of the answers of open questions are 
used in combining the results of the results of the closed questions and of the open questions. 

Optionally, answers given by respondents who saw other responses before answering 
are marked accordingly to prevent bias. Alternatively or additionally, the respondents are 
divided into different groups viewing controlled different stimuli, so as to control the bias and 
allow separate statistical analysis of different bias groups. 

In some embodiments of the invention, summary includes a comparison of results 
received for different tested concepts. Optionally, the comparison is shown in the form of a 
table or graph which shows a comparison of the results of same questions relating to the 
different concepts. Alternatively or additionally, for open questions, the summary points out 
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similar answers received for different concepts and optionally the number of times the similar 
answers appear for each concept. 

The summary may include an analysis of the results. In some embodiments of the 
invention, the results in the summary are analyzed to determine whether they achieve a 
5 predetermined goal, for example finding three best possibilities out of ten, or finding a best 
choice with at least a predetermined margin. The summary optionally states whether the goal 
was reached and/or suggests questions to be used to reach the goals. It is noted that by having a 
large number of respondents and using the tools provided by the present invention, knowledge 
of the goals and how close they are to being achieved may be used during the survey session in 
10 asking additional questions and/or utilizing additional respondents, so as to achieve the goals. 
Alternatively or additionally, when the goals are reached, the survey may be terminated before 
utilizing all the questions prepared in advance, so as not to waste time on questions which are 
not needed. 

In some embodiments of the invention, the summary includes results of a variance 

15 result analysis which searches for inter-related parameters. 

In some embodiments of the invention, in generating the summary, the answers are 
reviewed to weed out erroneous answers. The review is optionally performed automatically by 
station 110 and/or manually by human moderator assistants to which presentation station 110 
displays all or suspicious answers. For example, answers from a respondent who provides 

20 contradicting answers may be weeded out. Alternatively or additionally, the answers of a 
respondent who did not answer critical questions, did not answer a large number of questions 
and/or stated that he/she is not familiar with a discussed product, are weeded out. Alternatively 
to weeding out all the answers of a respondent, only the answers which are clearly 
contradicting or are clearly meaningless are weeded out. Further alternatively or additionally, 

25 lower weight is provided in the summary to answers from a respondent providing contradicting 
answers. In some embodiments of the invention, the roster includes duplicate questions 
(possibly using different wording) directed to determine the consistency of respondents, as is 
known in the art. 

Alternatively or additionally, in generating the summary, the answers are reviewed to 
30 weed out obscene statements. For example, the respondents may be provided with rating 
buttons to be used for rating remarks as inappropriate for display. Alternatively or additionally, 
the answers are compared to a dictionary of dirty words and answers including inappropriate 
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words are deleted or are provided to the moderator (or a moderator assistant) for examination 
before they are provided to other respondents. 

In some embodiments of the invention, obscene statements are not forwarded to other 
respondents. Alternatively or additionally, answers from respondents providing obscene 
statements are not provided to other respondents. In some embodiments of the invention, 
respondents providing obscene statements, having statements which tend to be weeded out 
and/or having a contradicting response pattern, are blocked out of the survey system. 

In some embodiments of the invention, each respondent has on his/her respective 
computer 104, controls for rating the answers and/or remarks of other respondents. The ratings 
optionally relate to agreement with the statement. Alternatively or additionally, the ratings 
relate to clarity, preciseness, importance and/or relevance of the answers and/or remarks. 

In some embodiments of the invention, the display of the answers to the moderator is 
affected by the ratings given to the answers. For example, the summary may highlight or 
enlarge those statements which have high and/or low levels of agreement and/or those remarks 
which are rated as clear. Alternatively or additionally, only answers of interest, e.g., receiving 
high levels of agreement and/or disagreement, are displayed to the moderator, so as not to 
flood the moderator with data. 

Optionally, when the survey session is part of a set of sessions of a single research 
(carried out in series and/or in parallel), the summary is displayed along with a comparison to 
previous sessions of the research. Alternatively or additionally, an accumulative summary is 
displayed. Optionally, the data of each session is displayed along with demographic 
information on the respondents of the session. Alternatively or additionally, the data of each 
session is displayed along with context information on the data the respondents saw. In some 
embodiments of the invention, moderator 128 may request to see the results given by 
respondents having one or more characteristics, e.g., age, gender, income. In some 
embodiments of the invention, the summary data of each session states bias related parameters, 
such as the data displayed to the respondents before answering the question. 

In some embodiments of the invention, a summary of discussions between respondents 

102 is also provided to moderator 128. Optionally, the summary includes only remarks, which 

are expected to be of interest. In some embodiments of the invention, the summary includes 

only short remarks, long remarks, or remarks which are otherwise expected to be of interest. 

Alternatively or additionally, the summary includes only remarks from specific respondents, 

for example leader respondents and/or respondents who are expected to provide unique 
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remarks. Optionally, the summary is created by presentation station 110 using text mining 
methods, based on ratings provided by other respondents and/or using a combination thereof. 

In some embodiments of the invention, the remarks are displayed chronologically 
according to the time the statements were entered by the respondents, as a sequence of text 
5 items. Alternatively or additionally, the moderator's display is divided into a plurality of 
frames (or windows, columns or other display division units) corresponding to different 
discussion groups and/or sub-groups of respondents. In each frame, the remarks and answers 
of a different group are displayed. Further alternatively or additionally, the moderator's display 
is divided into a plurality of frames, each of which displays data relating to a different context. 

10 In an exemplary embodiment of the invention, the display includes a first frame which shows a 
summary of all the data and a second from which shows details of a specific context, group, 
question or respondent of interest. 

In some embodiments of the invention, the moderator's display shows the questions 
provided to the respondents. Upon clicking on a question, the display shows the answers given 

15 to the question, or a summary of the answers. Upon clicking on an answer, the display 
optionally shows the remarks given to the answer (optionally in a tree or chain format), and so 
on. Alternatively to starting the display with questions, the display starts with showing 
concepts. In some embodiments of the invention, the moderator may control the organization 
of the display, for example sorting and/or filtering questions and/or answers according to 

20 timing, type and/or context. The sorting and filtering may be performed using various keys, 
such as the text of the statements, their ratings, the extent of additional discussion they 
spawned and/or the respondent names. 

In some embodiments of the invention, answers and/or remarks are displayed along 
with the ratings they received. Alternatively, only the last remark in a chain of remarks is 

25 displayed with a rating. 

In some embodiments of the invention, before the survey session, a set of goals is 
defined for the survey, for example in the form of a table. For example, the goals may include 
finding a concept with a maximal cost and purchase level, and finding out which factors affect 
purchaser's decisions. The summary optionally continuously and/or periodically updates the 

30 table of goals. The moderator optionally keeps track of the goals in determining the track of 

the survey and when to terminate the session. 

In some embodiments of the invention, the goals are defined in statistical terms, for 

example requiring a confidence level of 90 or 95% and with an error of± 5 or 10%. In some 
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cases, lower confidence levels, such as 70%, 75% or 80% may be sufficient. Another example 
of a definition is that of a statistical test, for example a "t" or "f ' test, which a survey result 
should satisfy to be acceptable. Another example is whether a small population sample is 
required or a large one. Another example is to reconstruct a type of a distribution of a variable 
(e.g., Gaussian or bi-modal) and/or other parameters, such as variance. 

Example of statistical goals for qualitative feedback, include, for example, for an open- 
ended question, a probability of over X% (e.g., 70%, 80%, 90%) that the Y answers selected 
cover the entire spectrum or at least Z% (e.g., 70%, 80%, 90%) of a spectrum. An example of 
a goal for a specific answer, that the answer represents the entire (or part, e.g., 30%, 50%, 
70%, 80%, 90%) of the population, with a certain confidence (e.g., 70%, 80%, 90%). If 
answers are coded and/or otherwise combined, various quantitative statistical tests can be 
applied. 

Referring in more detail to generating (214) additional questions based on the received 
answers, in some embodiments of the invention, presentation station 110 generates questions 
which probe the reasons for answers to previous questions. Optionally, the generated questions 
probe the reasons to a statistical distribution of the answers. For example, for an answer which 
was given by the most respondents and/or was given by a number of respondents above a 
threshold, a question asking why this answer is so popular, is generated. Alternatively or 
additionally, for answers given by very few respondents, a question asking why the answer was 
not selected, is generated. In some embodiments of the invention, a question asking if the 
question was clear is generated for questions having a low response rate. In some embodiments 
of the invention, when two answers receive substantially the same agreement levels, a question 
is formulated to determine which is preferred. Optionally, before formulating the question an 
attempt is made to ensure that the answers are not equivalent. In some embodiments of the 
invention, text mining methods are used to determine whether the answers are the same. 
Alternatively or additionally, the answers are displayed to the moderator who verifies that the 
answers are not equivalent. 

In some embodiments of the invention, when there is a statistically distinct difference 

between the answers of different groups (e.g., people of different ages, gender, professions), 

presentation station 110 generates questions probing the reasons of the difference ("why do 

people above age 50 prefer packaging in bags over cartons?"). Alternatively or additionally, 

when there is a big difference between the results of the current session and previous sessions, 

a question is generated to probe for the reasons. In some embodiments of the invention, when 
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respondents rate an answer of a different respondent as unclear, a question asking if they could 
rephrase the remark is generated. Alternatively or additionally, when respondents having a 
certain bias answer differently than others, a question asking whether the bias is the cause of 
the difference, is generated. 
5 Alternatively or additionally, presentation station 110 generates a closed question using 

answers given to an open question, for example the five most popular answers, the most 
unpopular answers and/or a combination of popular, unpopular and/or medially provided or 
rated answers. Alternatively or additionally, the answers used in the closed question are those 
provided by people with specific profiles, the shortest answers, answers having a correct 

10 grammatical structure and/or the first few answers entered. The popularity is optionally 
determined based on the ratings given to the answers and/or based on an accumulation of 
similar answers. The answers included in the closed question are optionally ones determined to 
be different from each other, for example using text mining methods. Alternatively or 
additionally, the answers included in the closed questions are those which are determined 

1 5 based on text mining methods as most differing. 

In some embodiments of the invention, presentation station 110 evaluates the attitude 
of the respondents to different types of questions, according to the answers of the respondents 
to previously provided questions. Accordingly, the questions in the roster of questions are 
optionally converted into a type more suited for the audience of respondents. The conversion 

20 may be performed in real time or may be performed by reference to a library of equivalent 
questions prepared before the survey session. Alternatively or additionally, the conversion is 
performed by requiring a moderator assistant to perform the conversion during the session. 
Optionally, at each time point, the moderator assistant performs the conversion of roster 
questions to be used in the next few moments. For example, if one type of question receives a 

25 low answer rate or receives superfluous, answers, while another type of question receives a 
high answer rate and/or detailed answers, questions of the first type are optionally converted 
into questions of the second type. 

The different types of questions may differ, for example, in their length, in using soft 
words (e.g., dislike) versus strong words (e.g., hate) and/or in the subject of the question (e.g., 

30 the respondent, friends, neighbors, people in general). For example, the same idea may be 
formulated using the following types of questions: 
What soap do you prefer? 
Which soap is best? 
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What soap do your friends prefer? 

Which soap do people buy? 
Questions may be provocative or polite, relate to past, present or future, etc. 

It is noted that each question type may involve a bias in a specific direction. 
5 Presentation station 1 10 optionally keeps track of which questions were used and accordingly 
analyzes the answers received. 

Optionally, a question type which receives best results from the respondents is used in 
a higher frequency than other types of questions. Alternatively or additionally, the questions 
are converted into different types, or the types are randomly selected, to avoid monotonous 
10 question asking. 

In some embodiments of the invention, the roster of questions prepared before the 
survey session includes questions in a canonical format. During the session, presentation 
station 1 10 converts (using any of the methods described above) each question being provided 
to the respondents into a format best suited the audience of respondents, according to 
15 information acquired during the session. Alternatively or additionally, the question types used 
are selected according to predetermined information on the respondents, such as demographic 
characteristics and/or answering capabilities. 

In some embodiments of the invention, the respondents may provide ratings on the 
questions. These rating are optionally displayed to moderator 128. Alternatively or 
20 additionally, based on the ratings, suggested changes to the questions of the roster and/or 
suggested new questions are provided. 

Optionally, in addition to questions generated by presentation station 110, questions 
received from observers 136 are displayed to moderator 128. In some embodiments of the 
invention, different observers 136 have different priorities and each suggestion is displayed 
25 before moderator 128 according to the priority of the observer providing the suggestion. 

In some embodiments of the invention, presentation station 110 prioritizes the 
displayed questions according to pre-configured rules. The more important questions may be 
displayed with a unique font attribute (e.g., size, style, color) and/or in a more central place. 
Alternatively, the questions are displayed in an equal manner without any prioritization 
30 provided by station 110. 

Referring in more detail to providing (216) answers to other respondents, in some 

embodiments of the invention, the respondents are divided into sub-groups, and the answers 

from the respondents of each sub-group are distributed between the respondents of that sub- 
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group. The sub-grouping is optionally performed in order to technically allow the discussion 
(not too many people talking together) and/or to stimulate a discussion of a specific blend of 
people. 

The sub-groups are optionally of a size which allows the respondents in the sub-group 
5 to apprehend the answers from all the other respondents of the sub-group. An exemplary sub- 
group size includes between about 10-20 respondents. In some embodiments of the invention, 
the respondents are divided into sub-groups randomly. Alternatively, the respondents are 
divided into sub-groups according to respondent profiles of the respondents. For example, 
slow readers may be collected together into a smaller group. In another example, respondents 
10 of same age groups may be grouped together. Alternatively or additionally, each sub-group 
includes a predetermined blend of respondents having different characteristics. The 
characteristics may include, for example, age, gender, economic status and/or other 
demographic characteristics. Alternatively or additionally, the characteristics may include 
verbosity, creativity, and/or other discussion related characteristics. 
15 Optionally, in some implementations, each respondent is only aware of his or her sub- 

group and does not know of other sub-groups being surveyed at the same time. 

In some embodiments of the invention, at the beginning of a survey session, 
preliminary questions are asked to determine respondent characteristics for the grouping. 
Alternatively or additionally, respondent characteristics are determined before the survey 
20 session, for example based on previous sessions and/or a form filled out prior to the session. 

Optionally, a respondent may be included in a plurality of sub-groups. For example, a 
respondent who is a fast reader and/or writer may be put in a few sub-groups. In some 
embodiments of the invention, the groups are made relatively small, such that each respondent 
may be included in two or three groups. When a respondent is included in a plurality of 
25 groups, each remark is sent to the group to which it belongs, optionally according to the 
answer or remark to which the entered remark relates. 

Rather than using relatively small sub-groups in which all the members participate in 
the discussion, in some embodiments of the invention, larger groups are defined, in which only 
some of the answers and remarks of the respondents are distributed to the other members. For 
30 example, groups of 100 members in which only answers from 20 respondents are distributed, 
may be used. Optionally, the distributed statements are generated by a group of randomly 
selected respondents. Alternatively, the distributed statements are generated by respondents 

identified as providing interesting answers, respondents having a high rate of answer 
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generation and/or respondents selected using any other methods. Alternatively or additionally, 
the distributed statements are selected according to the structure and/or contents of the 
statements, for example short statements. 

Alternatively to having fixed sub-groups throughout the session, in some embodiments 
of the invention, the sub-group division varies during the session, according to the progress of 
the sub-groups. For example, one or more respondents from highly active sub-groups may be 
transferred to a sub-group which is relatively inactive. The transfer is optionally performed 
between questions, so as not to interrupt in the middle of a discussion. Optionally, the transfer 
is performed without the respondents being notified of the transfer. Alternatively, the 
transferred respondents are notified that they are being transferred. 

In some embodiments of the invention, the division into sub-groups changes during the 
session according to the questions which are being answered. Optionally, for questions which 
are expected to arouse active discussions, smaller sub-groups are defined, than for questions 
which do not generally arouse discussion. Alternatively or additionally, the division into sub- 
groups may change due to changes in the profiles of the respondents and/or the data context to 
which the respondent was exposed. 

Having a large number of respondents participating in the discussion, allows the 
moderator to receive a large amount of qualitative input. It is noted, however, that there may 
not be a need for so much input on a single subject. Still, the large amount of respondents 
allows the moderator to receive required amounts of free expression input, even when the 
number of respondents from which input is required is not known in advance. In an exemplary 
embodiment of the invention, an important discussion group is defined by the moderator. The 
moderator follows the discussions in the important discussion group, while the remaining 
discussion groups are followed by observers 136 and/or their contents are stored for analysis 
after the survey session. Alternatively, the other respondents do not participate in discussions 
but rather listen to the discussion of the important group. The listeners may provide ratings to 
the statements made in the discussion, providing quantitative data on the remarks. 

In some embodiments of the invention, instead of transferring only a portion of the 
generated answers and remarks to other respondents, respondents may receive summaries as 
shown, for example, to the observers. 

The moderator optionally monitors the important discussion group, for example 
deciding who participates in the group and/or how many respondents are in the important 
discussion group. For example, when the discussion is not lively enough, more respondents 
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may be added to the group and/or one or more talkative people may be added to the group. 
Conversely, when the discussion is too dense, one or more respondents may be removed from 
the group. 

Alternatively to having sub-groups in which all respondents receive the remarks and 
5 answers of all the other respondents, in some embodiments of the invention, each respondent 
receives remarks and answers (referred to together as statements) from other respondents 
according to a personal respondent profile and/or a current state of the respondent. Optionally, 
the personal profile determines the amount of data received by the respondent. For example, 
the number of remarks and answers received may depend on the rate of writing and/or reading 
10 of the respondent. Alternatively or additionally, the personal profile determines the type of 
remarks to be provided to the respondent. For example, optimistic respondents may be 
provided with statements including negative words. In some embodiments of the invention, 
respondents are provided with statements which disagree with their opinion. Alternatively, 
respondents are provided with a blend of agreeing and disagreeing remarks according to the 
15 respondent's profile. In some embodiments of the invention, respondents receive all remarks 
which respond to answers that they generated. 

In some embodiments of the invention, respondents are provided with statements from 
people that have different opinions than they have. Optionally, respondents are identified as 
having different opinions according to the ratings they give. Respondents giving very different 
20 ratings to the answers they see are considered having different opinions. Alternatively or 
additionally, a respondent is considered having opposite opinions if the ratings given to the 
statements of the respondent have a high disagreement level. 

In some embodiments of the invention, when a group of respondents are determined to 
give similar ratings or otherwise have similar opinions, when one of the respondents of the 
25 group agrees with an answer, the others in the group are shown the opposite of the answer. 

Optionally, an attempt is made to give an equal exposure to each of the respondent's 
answers, i.e., each answer is shown to a predetermined number of other respondents. 
Alternatively or additionally, each answer is given at least a minimal exposure. 

In some embodiments of the invention, the number of remarks provided to each 
30 respondent depends on the time line of the respondent's session. Optionally, when a 
respondent is determined to be inactive, the respondent is provided with a relatively large 
number of answers from other respondents. An active respondent or a respondent having too 
much activity is optionally provided with fewer answers from other respondents. 
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Optionally, the time at which the respondent is allowed to see answers from other 
respondents depends on the respondent's profile. For example, a respondent determined to be a 
group leader and/or one who has independent thought, is optionally allowed to see answers 
from other respondents before answering questions. Follower respondents who tend to reiterate 
answers of other respondents are optionally shown answers from other respondents only after a 
predetermined amount of time for thought or after the respondent provides a first answer. 
Optionally, a minimum thought period is defined for one or more of the respondents, in which 
the respondent is not shown answers of other respondents. Answers of other respondents 
received during the minimal thought period are optionally accumulated and transmitted to the 
respondent together, after the minimal thought period. Alternatively, the accumulated answers 
are provided with a minimal period between the providing of the answers, allowing the 
receiving respondent to read, comment and/or rate the answers. Alternatively, answers and 
remarks from other respondents are delayed by a predetermined period, on their transmission 
to respondents who tend to reiterate answers of other respondents. Further alternatively, only 
responses which are not expected to fit the respondent's opinion are shown to the respondent 
before answering. For example, the respondent may be provided with comments from 
respondents who the respondent disagreed with their answers. 

In some embodiments of the invention, respondents are determined to be leaders 

according to the ratings their answers receive from other respondents. Alternatively or 

additionally, at the beginning of the session, a test is performed to determine whether the 

respondent is a leader or follower. Optionally, several test questions (e.g., 3-4) are provided to 

the respondents along with answers allegedly provided by other respondents. A leader score is 

given to the respondent according to the similarity between the respondent's answers and the 

answers provided to the respondent. Alternatively or additionally, the determination of whether 

a respondent is a leader is performed during the survey session, by comparing the respondent's 

answers to answers showed to the respondent. 

In an exemplary embodiment of the present invention, for each time a respondent 

repeats an answer of a different respondent, the time between providing a question and 

providing answers from other respondents is lengthened. For each time the respondent 

provides an original answer, the time between providing a question and providing answers 

from other respondents is shortened. Alternatively or additionally, the leader score is assigned 

to a respondent based on agreement ratings provided by other respondents. Further 

alternatively or additionally, the leader score of a respondent is a function of the average time 
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that the respondent requires in order to answer questions. A respondent who provides fast 
responses is optionally given a higher leader score. Further alternatively or additionally, the 
structure of the respondent's answers are analyzed in providing the leader score. For example, 
a respondent providing short and concise answers (e.g., including a single verb) are given 
5 higher leader scores. Alternatively or additionally, elaborate answers are given high leader 
scores. 

Alternatively or additionally, answers are provided to other respondents after a 
predetermined number of respondents answered the question. Optionally, the predetermined 
number of respondents answering the question includes a sufficient number that allows 

10 quantitative analysis. Thus, the fact that the other respondents may be biased from the 
questions does not eliminate the possibility of generating statistically based results. 

In some embodiments of the invention, respondent profiles also relate to the extent to 
which the respondent changes opinions. The change of opinion may be determined using the 
methods described above for finding leaders. 

15 In some embodiments of the invention, presentation station 110 is pre-programmed 

with distribution rules which govern the distribution of data. Optionally, during the survey 
session, moderator 128 may change one or more rules if desired. In some embodiments of the 
invention, presentation station 110 includes a rule interface which allows moderator 128 to 
enter changes in the rules in a fast manner. Optionally, a plurality of sets of distribution rules 

20 are defined and moderator 128 selects one of these sets of rules to be used. Alternatively or 
additionally, for each question and/or context, moderator 128 selects a set of rules to be used, 
optionally if a default set is not desired. 

Optionally, the moderator and/or the observers may provide remarks and/or answers as 
if they are coming from respondents. This may be used, for example, to stimulate discussion. 

25 Referring in more detail to respondents providing (218) remarks to other answers, in 

some embodiments of the invention, the remarks are free text statements of a free discussion. 
Alternatively or additionally, the remarks include ratings, for example agree/disagree and/or on 
a scale of 1-5 or 1-10. Optionally, respondents may provide remarks to remarks of other 
respondents. In some embodiments of the invention, the remarks are provided in a free 

30 discussion manner. Alternatively, remarks are allowed only in direct response to answers. 

Further alternatively or additionally, remarks are allowed in response to answers or remarks 

directly related to answers. Optionally, a remark is considered directly related to an answer if 

the number of remarks connecting the remark to the answer is beneath a predetermined 
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threshold (e.g., between 1-3). Further alternatively, remarks may be made without specific 
relation to a different answer or remark. 

In some embodiments of the invention, at any time, a respondent may provide a remark 
only to the most current question, answer, and/or remark he/she is displayed. Alternatively, the 
respondent may relate to one or more previous statements. Optionally, remarks may be entered 
only with relation to a current context. 

Optionally, computer 104 records the statement to which the remark relates and the 
remark is transmitted to presentation station 1 10 together with an indication of the statement to 
which it relates. In some embodiments of the invention, the summary displayed to moderator 
128 shows the relation between the remarks provided by the respondents, for example in a tree 
structure. Optionally, each remark displayed to the moderator is accompanied by the statement 
to which the remark relates. Alternatively or additionally, each remark displayed to the 
moderator is accompanied by information on the context of the originator of the remark. The 
context optionally includes information on the data provided during the survey session to the 
originator of the remark and/or information on data generated by the originator of the remark 
during the survey session. Further alternatively or additionally, in order to reduce the amount 
of data displayed to the moderator, the information on the context is displayed in an abridged 
format. The abridged format may state a number of data pieces provided to the originator 
and/or a number of stimulus provided to the originator. 

Optionally, the distribution of the remarks depends on the statement to which the 
remarks relate. In some embodiments of the invention, remarks are always provided at least to 
the generator of the statement to which they relate. Alternatively or additionally, in 
embodiments in which not all respondents are in the same context at the same time, statements 
are distributed only to respondents 102 currently in the same context as the statement to which 
the remark relates. 

In some embodiments of the invention, presentation station 110 controls for each 
statement provided to a respondent, the input that the respondent may provide on the statement 
and/or must provide for the statement. The input may include, for example, remarks, ratings 
and/or no input at all. When a respondent must enter a response, computer 104 may not allow 
the respondent to proceed without entering the response or may repeatedly remind the 
respondent that the response is due. Alternatively or additionally, computer 104 warns the 
respondent that a fine will be deducted from the compensation of the respondent if a response 
is not provided. 

41 


WO 2004/042492 


PCT/BL2003/000923 


Referring in more detail to deciding (220) when to end the discussion on the cun-ent 
question, in some embodiments of the invention, the moderator follows the discussion and 
accordingly determines when to move to end the discussion of the present question. 
Alternatively or additionally, presentation station 110 determines when the discussion 
5 dwindles, for example when only few remarks are received. Further alternatively or 
additionally, presentation station 110 determines when the remarks got carried away from the 
main subject, for example by identifying that key words representative of the question do not 
appear in the discussion. According to the determination of presentation station 110 the 
moderator determines when to proceed to a next question. Alternatively, presentation station 

10 110 ends the discussion of the question automatically. 

In some embodiments of the invention, each question is assigned a predetermined 
amount of time for discussion and when the time is exhausted, the discussion is terminated. 
Some questions (e.g., closed questions) may, optionally, be given no time for discussion. 
Optionally, if the discussion dwindles before the time is over, the discussion is terminated 

15 early. In such cases, the remaining time is optionally allotted as additional time to other 
questions or to additional questions added in real time by the moderator. Optionally, closed 
questions are given less time for discussion than open questions or are not given any 
discussion time at all. 

In the above discussion it was assumed that all respondents move to a new question at 

20 the same time. In some embodiments of the invention, however, different respondents may 

have different question time lines. Optionally, each respondent proceeds through the questions 

of the roster at his/her own rate according to the rate in which the respondent responds to 

questions, the amount of data the respondent receives from other respondents and/or the 

participation of the respondent in discussions with other respondents. 

25 In some embodiments of the invention, constraints are placed on the proceeding of 

respondents. For example, respondents may be prevented from proceeding beyond a 

predetermined point of the question roster, for example a concept change point, before the 

other respondents reach the point. Optionally, for each point at which respondents may be 

stalled, the roster includes filler questions to be provided to advanced respondents. Optionally, 

30 the filler questions are prompts for remarks on answers of other respondents. In some 

embodiments of the invention, only fast respondents are allowed to provide remarks on other 

respondents answers, while other respondents may only give ratings or may not refer at all to 

answers of other respondents. In some embodiments of the invention, when a predetermined 
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number (e.g., 90%) of the respondents reach the change point, slow respondents are moved 
beyond the point, foregoing their answers to the skipped questions. Alternatively or 
additionally, respondents are prevented from proceeding at a rate substantially different from 
other respondents in a sub-group to which they belong. 

Alternatively to placing constraints, each respondent may proceed at his/her own pace 
even if this results in different respondents relating to different contexts at the same time. 

In some embodiments of the invention, remarks are passed only between respondents 
relating to the same question, such that discussions are always linked to a single question. 
Alternatively, remarks are passed between respondents relating to questions of a same context. 
Further alternatively or additionally, remarks may be passed also to respondents in other 
contexts, optionally provided they already answered questions of the context to which the 
remarks relate. 

In some embodiments of the invention, the moderator may define for some or all of the 
respondents and/or for one or more of the questions, that the respondent cannot participate in 
the discussion of the question before submitting an answer to the question. Alternatively or 
additionally, a respondent may be defined not to receive a next question before answering a 
previous question. Optionally, one or more questions are defined as less important. If a 
respondent is behind other respondents these questions are omitted in order to allow the 
respondent to catch up to other respondents. 

Optionally, the moderator may provide a question which interrupts the time line of all 
the respondents. After answering the question, the respondents may return to their time lines, 
or may continue together along a new time line. 

In some embodiments of the invention, each respondent receives data at a rate (or 
average rate) determined for the respondent. Questions, answers and remarks directed to the 
respondent are optionally collected in a buffer of the respondent in presentation station 110 
and/or on the respective computer 104. Data from the buffer is optionally displayed to the 
respondent at the determined rate. Optionally, the contents of the buffer are displayed to the 
respondent in the order of reception. Alternatively or additionally, questions and/or remarks 
from specific other respondents are given priority in being displayed. In some embodiments of 
the invention, the rate at which data is provided to the respondent depends on the amount of 
data in the buffer of the respondent. 

Referring in more detail to determining whether (222) to provide an additional 
question, in some embodiments of the invention, the determination is performed according to 
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whether all the questions in the roster of questions were provided to the respondents. 
Alternatively or additionally, the survey session is allotted a predetermined amount of time and 
at the end of this time the session is terminated. 

Referring in more detail to selecting (224) an additional question by the moderator, 
presentation station 110 optionally displays suggested questions and the responses from the 
respondents in a manner which allows fast decision making by the moderator. The additional 
question may be selected from the roster, may be adopted from questions suggested by 
observers and/or by presentation station 1 10 and/or may be suggested by the moderator. 

It is noted that the moderator may identify an important point in the discussion of one 
of the groups, formulate a question based on the identified point and provide the question to all 
the respondents. In some embodiments of the invention, however, the question may be 
provided to all the respondents except for those in the sub-group from which the idea of the 
question was taken, as the respondents in this sub-group may be biased. 

In some embodiments of the invention, all the respondents receive the same question at 
the same time. Thus, complete results of the question may be provided substantially 
immediately. Alternatively, different respondents may receive different questions at the same 
time and/or the same question may be provided to different respondents at different times. For 
example, the moderator may instruct presentation station 1 10 to provide a question to each of 
the respondents. Presentation station 110 optionally provides the question to the buffer of each 
of the respondents, but the actual delivery time changes with time. 

In some embodiments of the invention, the providing of questions is performed by the 
moderator, while answers and remarks are transferred automatically based on rules governing 
the transfer. Alternatively or additionally, the moderator may have the questions provided 
automatically by presentation station 110, according to a predetermined plan, in a manner 
similar to the transfer of answers. For example, presentation station 110 may be instructed to 
provide the questions from the roster at a predetermined rate. Optionally, the moderator may 
interrupt in the automatic provision of questions, when desired. Alternatively or additionally, 
the predetermined plan may include points in which the moderator is prompted for a decision, 
for example when two different questions were prepared in advance or when it is expected that 
additional probing will be required based on the responses. 

The plan for providing questions may be, for example, based on a simple time line 
and/or may depend on other parameters, such as the answers provided by the respondents. 
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In some embodiments of the invention, the moderator may determine to provide a 
question only to a portion of the respondents. For example, in order to save time, half the 
respondents may be given a first question while the other half of the respondents receive a 
second question. Based on the results from the respondents to which the question was 
provided, the moderator may decide if to provide the question to the additional respondents. 
For example, a question receiving feedback that it is unclear or non-interesting may be 
removed from being provided to other respondents. 

In some embodiments of the invention, each question in the roster and optionally those 
provided by the moderator during the survey session, is associated with an indication of its 
minimal exposure needs. The exposure needs may include the number of people to which the 
question is to be provided and/or the demographic, geographic and/or other characteristics of 
the people to receive the question. For example, a question may be marked as requiring at least 
100 respondents, with a blend of 70% men and 30% women. Optionally, when a question is 
created it may be assigned a default exposure need indication, for example at least 80% of the 
respondents. Alternatively or additionally, questions may be assigned with a required statistical 
confidence level. 

During the session, the delivery of the questions to the respondents (including which 
respondents receive each question) is optionally performed automatically by presentation 
station 110, based on the exposure needs of the question. Alternatively or additionally, 
questions are associated with a minimal number of responses required. The question may be 
provided to a predetermined number of respondents and if not enough responses are received 
more respondents are provided with the question. 

In some embodiments of the invention, some respondents may be provided with 
questions which they are not allowed to answer or they are not required to answer. The display 
of the question may aid them in participating in a discussion with other respondents. 

The question displayed to one or more of the respondents may be adapted to the profile 
of the respondent. In an exemplary embodiment of the invention, some respondents are 
provided with closed questions, while other respondents receive half closed questions. 

In some embodiments of the invention, the respondents are divided into sub-groups 

which relate to the questions of the roster in different orders, in order to cancel the effect of the 

order of exposure on the results. For example, if the survey is directed to evaluating five 

concepts, the respondents may be divided into five groups, each of which begins its discussion 

with a different concept. Presentation station 110 optionally automatically governs the 
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provision of questions to each of the groups. Moderator 128 optionally follows the group 
discussions and accordingly may add/change or remove questions in each of the concepts. 
Optionally, the change of a question is affected in all the groups which did not already 
complete the discussion of the change concept. In some embodiments of the invention, the 
survey session concludes with a series of closed and/or half closed questions which summarize 
the survey, allowing the moderator to provide questions based on the discussion, to all the 
respondents. 

In some embodiments of the invention, the discussion summary provided to the 
moderator states for each respondent the concepts to which the respondent was exposed. 
Alternatively or additionally, in order to simplify the information provided to the moderator, 
the summary states for each respondent the number of concepts to which the respondent was 
exposed, and/or which of a predetermined number of general flow threads the respondent 
followed. It is noted that some respondents may be moved between groups during the session, 
and therefore the information on the information already viewed by the respondent may be 
important to understanding the respondents remarks. In some embodiments of the invention, 
the movements of respondents between sub-groups may be limited to specific flow threads, in 
order not to have too many possible contexts of the respondents. 

The above incorporated PCT publication WO 01/84266 described how to selectively 
expose information to participants in a mass interaction event. In an exemplary embodiment of 
the invention, those methods or the methods described herein are used. However, in an 
exemplary embodiment of the invention, the survey system distributes data not only based on 
the load of the recipients, but, in some cases, instead or in addition, taking into account 
statistical requirements and/or a bias of a single (or several) respondent, a (or several) group, a 
(or several) session and/or a complete survey. 

In one example, if a respondent can only tolerate three out of four data items, a test 
(e.g., statistical analysis) is made to see which one should be dropped, by checking for each 
one if dropping it will, for example, cause bias, and/or reduce a confidence level below a 
defined threshold. In another example, data is not forwarded if a statistical analysis shows that 
a comment from a respondent receiving the data will not significantly improve a statistical 
requirement or if a goal can be reached without sending the data. 

In an exemplary embodiment of the invention, the survey system can also provide a 

"what-if ■ functionality, which can, for example, be applied to see if changing data distribution 

rules in a certain way is likely to affect a statistical confidence level. In an exemplary 
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embodiment of the invention, the "what-if ' functionality is supported by simulations which 
model the activities of respondents under certain assumptions. These assumptions may be 
changed and/or learned based on the behavior of the respondents so far. In an exemplary 
embodiment of the invention, the system applies a general what-if scenario to ensure that the 
survey will not fail, for example, maintaining a cache of available unbiased respondents and 
ensuring that required goals are met with some leeway, for example so that if a particular 
respondent leaves suddenly, or otherwise disrupts the survey, meaningful results can still be 
achieved. Reduced backup goals are optionally defined for such cases. 

Referring in more detail to generating (226) a session report, in some embodiments of 
the invention, the session report includes quantitative data on the answers of the respondents to 
at least one of the questions. Optionally, the quantitative data includes statistics from ail the 
respondents or substantially all the respondents. The report optionally includes for each 
question a percentage of respondents giving each of a predetermined number of answers. 
Alternatively or additionally, the report includes average ratings of one or more questions. 
Optionally, the report also includes representative remarks made by respondents. Optionally, 
any of the methods described above for providing the summary to the moderator may be used 
in generating the report. 

In some embodiments of the invention, the report includes ratings provided by the 
respondents. Alternatively or additionally, the report includes a comparison of various 
concepts and/or of different demographic groups. 

In some embodiments of the invention, the report includes explanations given by the 
respondents to the quantitative data. Optionally, the report further includes quantitative data on 
the explanations given to the quantitative data. 

It is noted that although the above embodiments are described as utilizing sophisticated 
communication methods, in some embodiments of the invention, the discussions are 
performed using relatively simple communication methods, such as chat rooms and/or 
clipboards. For example, each sub-group may be implemented by a chat room, while closed 
questions are provided separately. 

Although the above description relates to a single moderator, in some embodiments of 
the invention, a survey session may be moderated by a plurality of moderators connected to 
different presentation stations 110. Optionally, each of the moderators has a different task. For 
example, the moderators may control different groups. Alternatively or additionally, one 

moderator controls the provision of questions while another moderator controls the transfer of 
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remarks. Further alternatively or additionally, the moderators have at least partially 
overlapping authority. Optionally, when contradicting instructions are received from two 
moderators, preference is given to instructions of one of the moderators according to a 
predetermined hierarchy. Alternatively or additionally, the instructions are passed to an 
arbitrating moderator. 

Alternatively to having a moderator, the survey session is governed completely 
automatically by presentation station 1 10 (which therefore does not need to have a display). In 
this alternative, in addition to suggesting questions, station 110 selects the questions to be 
provided. When a moderator 128 does manage the session, presentation station 110 may be 
used at different levels of aid and/or control. In some embodiments of the invention, 
presentation station 110 interprets general rules of the moderator into specific instructions 
passed to each of the computers 104. Alternatively or additionally, presentation station 110 
presents the moderator with information on the consequences of change in a specific rule, for 
example displaying the number of respondents to be affected. In some embodiments of the 
invention, moderator intervention is limited to high level rules. Alternatively, if desired, the 
moderator may give instructions for specific respondents. Optionally, a user friendly interface 
is used to allow the moderator to insert such instructions. 

It is noted that although the above described system allows simple management of 
large groups, including definition of sub-groups, in some cases it may be desired to break a 
survey session into separate sessions. This may be done before the survey session begins 
and/or during the survey session. Optionally, presentation station 110 may be instructed to 
convert a current sub-group distribution into fixed groups and/or into separate session groups. 

Although the above system is described as operating in an on-line real time setup, the 
invention may also be used in a non-real time set-up, for example a message board setup (e.g., 
each group or subgroup can have its own message board, which may be linked together by a 
moderator), in which each respondent periodically logs-in and provides input. In these 
embodiments of the invention, presentation station 1 10 may be programmed to alert the 
moderator to login when sufficient data has been accumulated. The alert may be sent, for 
example, by a telephone call, SMS message and/or an e-mail message. In some embodiments 
of the invention, the message board used is controlled, e.g., the respondent cannot view 
specific zones of the message board until providing an answer. Similarly, the other constraints 
described above may also be imposed in the message board environment. 
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Optionally, in these embodiments, the summary provided to the moderator may be 
more elaborate than for real time survey sessions. Alternatively or additionally, respondents 
may be instructed to give more elaborate answers. In some embodiments of the invention, 
when respondents login they are provided with a message board for review. Optionally, if 
other respondents belonging to a same sub-group are logged in at the same time, the 
respondents may be given a chance to discuss things on-line as in a chat room. Alternatively, 
chat discussions are allowed only if they do not cause bias, for example only if all the 
respondents of a group are on-line at the same time. Further alternatively, respondents do not 
login but rather respond using e-mail. 

It is noted that some of the methods described above for handling one or more types of 
statements (e.g., remarks, questions, answers) may be used for other types of statements. 

In an exemplary implementation of the invention, a survey is conducted for a shampoo 
on a sample of 300 participants. A goal is set that the achieved results have a 90% confidence 
on a sample of 80% women and 20% men. A roster of questions is generated for four different 
shampoo concepts, optionally substantially the same questions being asked for each of the 
concepts, including various comparison questions. Some of the comparison questions are 
optionally generated as templates into which two concepts having similar answer levels are 
later inserted. The roster optionally includes a few opening and/or warming up questions, 
possibly followed by demographic questions. Thereafter, the roster includes a series of closed 
questions for each concept, such as "how unique is concept X?", "how believable is concept 
X?", "how likely would you be to purchase concept X?", "how relevant is concept X to you?". 
In addition, the roster optionally includes open questions, such as "what do you like about 
concept X?", "what do you dislike about concept X?". The roster further includes instructions 
on the transfer of answers to the open questions from men to women, for rating. Alternatively 
or additionally, sub-groups of women are set to freely discuss each of the concepts, with an 
unlimited amount of remark transfer. Optionally, random remarks from the discussion are 
selected and transferred to the men for rating. 

Optionally, the roster includes one or more probing points at which the moderator is 
prompted to ask additional questions based on the data corrected until the probing point. 

In preparing the survey, the moderator optionally defines a method to choose and/or 

rate the concepts. For example, a purchase intent score for "definitely would buy" and 

"probably would buy" is defined as a most important factor, followed by a uniqueness level 

which may be used as a tie breaker and/or as a secondary condition. After a best concept is 
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chosen, questions are provided to determine why the concept is best and/or how the concept 
can be enhanced. 

The participants are optionally divided into four sessions of 75 participants or into four 
sub-groups. In each session, the questions of the roster are presented in a different order of 
concepts in order to cancel bias. Optionally, brand users and non brand users are identified, 
and different weights are given to their answers. 

During each session, the moderator exposes (displays) an image of each concept on 
computers 104 and asks the questions from the roster. The moderator may change the 
questions (generally slightly) based on the discussion and/or statistical results of closed 
questions. Alternatively or additionally, the moderator may add or remove questions from the 
roster. A brand manager of the shampoo, serving as observer 136, may optionally ask the 
moderator to add/modify questions based on the answers, statistical analysis of the answers 
and/or the remarks. 

Throughout the sessions, presentation station 110 optionally displays the answer 
statistics to the moderator. Optionally, when results from other sessions are available, the 
results from all the sessions are displayed accumulatively and/or comparatively. 

Optionally, after the second session, the research team decides to drop one concept that 
was rejected in the first two sessions and replace it with a new concept. Therefore the new 
concept will have only a total sample of 150 respondents and be somewhat less accurate in 
terms of statistical validity. 

It is noted that acts described above as being performed by moderator 128 may also be 
performed automatically based on pre-configured rules. Optionally, the moderator may 
interrupt the automatic operation when desired. Furthermore, acts described above as being 
performed automatically, may optionally be performed by moderator 1 28. Some of the tasks of 
the moderator are relatively trivial and may be easily delegated to assistants, so that the 
moderator can concentrate on the main task of controlling the interactive survey session. 
Optionally, the tasks passed on to moderator assistants, if at all, are chosen according to the 
expected or actual load on the moderator. 

It will be appreciated that the above described methods and systems of performing 

surveys may be varied in many ways, including, changing the order of steps performed. In 

addition, a multiplicity of various features, both of method and of apparatus have been 

described. It should be appreciated that different features may be combined in different ways. 

In particular, not all the features shown above in a particular embodiment are necessary in that 
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embodiment or in every similar embodiment of the invention. Further, combinations of the 
above features are also considered to be within the scope of some embodiments of the 
invention. Also within the scope of the invention are hardware and/or software combinations 
and/or computer readable media containing software for carrying out the methods described 
above. Section heads are provided solely to assist in browsing the application and should not 
be construed as limiting the applicability of a feature described therein to only that particular 
section. When used in the following claims, the terms "comprises", "includes", "have " and 
their conjugates mean "including but not limited to". 
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